Adding the pipeline for the task explanation and Llm #2190

Bepitic · 2024-07-15T14:45:31Z

📝 Description

I added the Task explanation and built the pipeline for the future LLM that will categorize and explain the image.
I modified the visualization such the explanation can be seen in the image outputted.

✨ Changes

Select what type of change your PR is:

🐞 Bug fix (non-breaking change which fixes an issue)
🔨 Refactor (non-breaking change which refactors the code base)
🚀 New feature (non-breaking change which adds functionality)
💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
📚 Documentation update
🔒 Security update

✅ Checklist

Before you submit your pull request, please make sure you have completed the following steps:

📋 I have summarized my changes in the CHANGELOG and followed the guidelines for my type of change (skip for minor changes, documentation updates, and test enhancements).
📚 I have made the necessary updates to the documentation (if applicable).
🧪 I have written tests that support my changes and prove that my fix is effective or my feature works (if applicable).

For more information about code review checklists, see the Code Review Checklist.

Signed-off-by: bepitic <[email protected]>

Bepitic · 2024-07-15T15:04:59Z

Hello, I'm looking to make this contribution as effective as possible, so please feel free to share any suggestions, concerns, or ideas you might have.
Thanks in advance for your time and input!

samet-akcay

added my initial comments, will do another round of review

src/anomalib/__init__.py

src/anomalib/callbacks/metrics.py

codecov · 2024-07-16T11:40:14Z

Codecov Report

Attention: Patch coverage is 63.49206% with 23 lines in your changes missing coverage. Please review.

Project coverage is 80.67%. Comparing base (e7e6cb8) to head (a463b5b).
Report is 2 commits behind head on main.

Files	Patch %	Lines
src/anomalib/utils/visualization/image.py	47.61%	22 Missing ⚠️
...anomalib/deploy/inferencers/openvino_inferencer.py	83.33%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2190      +/-   ##
==========================================
- Coverage   80.78%   80.67%   -0.12%     
==========================================
  Files         248      248              
  Lines       10859    10885      +26     
==========================================
+ Hits         8772     8781       +9     
- Misses       2087     2104      +17

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

…ipeline

Signed-off-by: Bepitic <[email protected]>

djdameln

Thanks for the PR.

I think it would be good to add the OpenAI model implementation to this PR, so that we can run the pipeline and generate some visualizations.

src/anomalib/__init__.py

src/anomalib/callbacks/metrics.py

ashwinvaidya17

The PR looks good but I am not sure about the expected output. Maybe it would be a good idea to add tests for this is possible.
This is the code I used for visualization. Is this what you would expect? (ignore the noisy image)

from anomalib.utils.visualization.image import ImageResult, ImageVisualizer, VisualizationMode
import numpy as np
from anomalib import TaskType
import matplotlib.pyplot as plt

anomaly_map = np.zeros((100, 100), dtype=np.float64)
anomaly_map[10:20, 10:20] = 1.0
gt_mask = np.zeros((100, 100))
gt_mask[15:25, 15:25] = 1.0
image = np.random.randint(0, 255, size=(100, 100, 3), dtype=np.uint8)
image_result = ImageResult(
    image=image,
    pred_score=0.9,
    pred_label="abnormal",
    text_descr=("Some very long text to see how it is formatted in the image"
                " Some very long text to see how it is formatted in the image"),
    anomaly_map= anomaly_map,
    gt_mask = gt_mask,
    pred_mask = anomaly_map,
)

image_visualizer = ImageVisualizer(
    mode = VisualizationMode.FULL,
    task= TaskType.EXPLANATION,
)
result = image_visualizer.visualize_image(image_result)

plt.imshow(result)
plt.show()

src/anomalib/utils/visualization/image.py

Signed-off-by: bepitic <[email protected]>

Signed-off-by: Bepitic <[email protected]>

openai. Signed-off-by: Bepitic <[email protected]>

Signed-off-by: Bepitic <[email protected]>

Co-authored-by: Dick Ameln <[email protected]>

Co-authored-by: Ashwin Vaidya <[email protected]>

Signed-off-by: Bepitic <[email protected]>

Co-authored-by: Samet Akcay <[email protected]>

Signed-off-by: Bepitic <[email protected]>

GPTVad

src/anomalib/__init__.py

ashwinvaidya17

Thanks! Looks fine overall. I just have a few minor comments.

tests/integration/model/test_models.py

src/anomalib/models/image/gptvad/lightning_model.py

tests/unit/models/image/chatgpt_vision/test_api.py

src/anomalib/models/image/gptvad/chatgpt.py

src/anomalib/models/image/gptvad/readme.md

Co-authored-by: Ashwin Vaidya <[email protected]>

Signed-off-by: Bepitic <[email protected]>

ashwinvaidya17

Thanks looks good to me

Signed-off-by: Bepitic <[email protected]>

…ipeline Signed-off-by: Bepitic <[email protected]>

Bepitic added 3 commits July 15, 2024 15:47

Add Task EXPLANATION and the visualization of images with description.

adbca17

Signed-off-by: bepitic <[email protected]>

upd dataset task with explanation

5611ec1

Signed-off-by: bepitic <[email protected]>

fix tasktype on metrics, depth, cataset, inferencer.

8ed23a3

Signed-off-by: bepitic <[email protected]>

Bepitic requested review from samet-akcay, ashwinvaidya17 and djdameln as code owners July 15, 2024 14:45

Merge branch 'main' into llm-pipeline

a463b5b

samet-akcay reviewed Jul 16, 2024

View reviewed changes

src/anomalib/__init__.py Outdated Show resolved Hide resolved

src/anomalib/callbacks/metrics.py Outdated Show resolved Hide resolved

src/anomalib/callbacks/metrics.py Outdated Show resolved Hide resolved

Bepitic added 7 commits July 16, 2024 23:42

fix lint on visualization/image

d5baf6b

Merge branch 'openvinotoolkit:main' into llm-pipeline

b7c8eaa

Merge branch 'llm-pipeline' of github.com:Bepitic/anomalib into llm-p…

5b563d9

…ipeline

Fix formatting dataset

bfd936e

fix format data/base/depth

f541316

Signed-off-by: Bepitic <[email protected]>

Fix formatting openvino_inferencer

4e392a9

Signed-off-by: Bepitic <[email protected]>

fix formatting

5fc70ba

Signed-off-by: Bepitic <[email protected]>

Bepitic requested a review from samet-akcay July 18, 2024 16:00

djdameln reviewed Jul 24, 2024

View reviewed changes

src/anomalib/__init__.py Outdated Show resolved Hide resolved

src/anomalib/callbacks/metrics.py Outdated Show resolved Hide resolved

ashwinvaidya17 reviewed Jul 24, 2024

View reviewed changes

src/anomalib/utils/visualization/image.py Show resolved Hide resolved

src/anomalib/utils/visualization/image.py Show resolved Hide resolved

Bepitic and others added 10 commits August 2, 2024 06:59

Add Explanation to error-msg.

75099af

Signed-off-by: bepitic <[email protected]>

OpenAI - VLM init

e5040d3

Signed-off-by: Bepitic <[email protected]>

Add wrapper to run OpenAI

86ad803

Signed-off-by: Bepitic <[email protected]>

add in ppyproject

3678f72

openai. Signed-off-by: Bepitic <[email protected]>

Add Test and fix description/title

7413842

Signed-off-by: Bepitic <[email protected]>

Add Readme and fix bug.

dc42cbd

Signed-off-by: Bepitic <[email protected]>

Update src/anomalib/models/image/openai_vlm/lightning_model.py

5788d22

Co-authored-by: Dick Ameln <[email protected]>

Update src/anomalib/models/image/openai_vlm/__init__.py

e4f6bec

Co-authored-by: Ashwin Vaidya <[email protected]>

Add fix pipeline bug.

5437467

Signed-off-by: Bepitic <[email protected]>

Add test.

982c9ca

Signed-off-by: Bepitic <[email protected]>

Bepitic requested review from ashwinvaidya17 and djdameln August 17, 2024 07:36

Bepitic and others added 11 commits August 20, 2024 13:41

Update src/anomalib/models/image/chatgpt_vision/__init__.py

224118b

Co-authored-by: Samet Akcay <[email protected]>

Update src/anomalib/models/image/chatgpt_vision/chatgpt.py

b703a41

Co-authored-by: Samet Akcay <[email protected]>

Update src/anomalib/models/image/chatgpt_vision/lightning_model.py

24c5486

Co-authored-by: Samet Akcay <[email protected]>

Update tests/integration/model/test_models.py

68e757e

Co-authored-by: Samet Akcay <[email protected]>

Update src/anomalib/models/image/chatgpt_vision/lightning_model.py

86714a1

Co-authored-by: Samet Akcay <[email protected]>

Update src/anomalib/models/image/chatgpt_vision/lightning_model.py

196d2a3

Co-authored-by: Samet Akcay <[email protected]>

fix comments

b7f345a

Signed-off-by: Bepitic <[email protected]>

remove last file of chatgpt_vision.

b285d10

Signed-off-by: Bepitic <[email protected]>

fix tests

a688530

Signed-off-by: Bepitic <[email protected]>

Merge pull request #1 from Bepitic/OpenAI-VLM (GPTVad)

0fb5f79

GPTVad

Merge branch 'main' into llm-pipeline

6503543

samet-akcay reviewed Aug 20, 2024

View reviewed changes

src/anomalib/__init__.py Outdated Show resolved Hide resolved

ashwinvaidya17 requested changes Aug 21, 2024

View reviewed changes

Bepitic and others added 8 commits August 21, 2024 12:32

Update src/anomalib/models/image/gptvad/chatgpt.py

8e92e5e

Co-authored-by: Ashwin Vaidya <[email protected]>

upd: language -> VISUAL_PROMPTING

5ab044d

Signed-off-by: Bepitic <[email protected]>

fix visual prompting and model_name

3f9ca93

Signed-off-by: Bepitic <[email protected]>

fix GPT for Gpt and the folder of the tests.

391b4c4

Signed-off-by: Bepitic <[email protected]>

fix: change import error outside.

ca1a0bb

Signed-off-by: Bepitic <[email protected]>

fix readme pointing to the right model.

022dcb7

Signed-off-by: Bepitic <[email protected]>

fix import cycle, and separate usecase by explicit if.

af7b9e9

Signed-off-by: Bepitic <[email protected]>

upd: add comments to the few shot / zero shot.

faf334f

Signed-off-by: Bepitic <[email protected]>

Bepitic requested a review from ashwinvaidya17 August 21, 2024 13:46

fix: dataset expected colums

3ed8d3f

Signed-off-by: Bepitic <[email protected]>

ashwinvaidya17 approved these changes Aug 21, 2024

View reviewed changes

Bepitic added 4 commits August 22, 2024 17:21

upd: add the same logic of the label on visualize_full.

7f454c4

Signed-off-by: Bepitic <[email protected]>

Merge branch 'main' into llm-pipeline

45bd520

Fix in the logic of the code.

44586d6

Signed-off-by: Bepitic <[email protected]>

Merge branch 'llm-pipeline' of github.com:Bepitic/anomalib into llm-p…

7adb835

…ipeline Signed-off-by: Bepitic <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding the pipeline for the task explanation and Llm #2190

Adding the pipeline for the task explanation and Llm #2190

Bepitic commented Jul 15, 2024

Bepitic commented Jul 15, 2024

samet-akcay left a comment

codecov bot commented Jul 16, 2024

djdameln left a comment

ashwinvaidya17 left a comment

ashwinvaidya17 left a comment

ashwinvaidya17 left a comment

Adding the pipeline for the task explanation and Llm #2190

Are you sure you want to change the base?

Adding the pipeline for the task explanation and Llm #2190

Conversation

Bepitic commented Jul 15, 2024

📝 Description

✨ Changes

✅ Checklist

Bepitic commented Jul 15, 2024

samet-akcay left a comment

Choose a reason for hiding this comment

codecov bot commented Jul 16, 2024

Codecov Report

djdameln left a comment

Choose a reason for hiding this comment

ashwinvaidya17 left a comment

Choose a reason for hiding this comment

ashwinvaidya17 left a comment

Choose a reason for hiding this comment

ashwinvaidya17 left a comment

Choose a reason for hiding this comment